Hi,

I am trying to create an instance of a BizTalk Pipeline object from within a
..net method.

The code would be something like this:
Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMes sages pipeline =
PipelineFactory.CreateEmptyReceivePipeline();
pipeline.AddComponent(new BtfDasmComp(), PipelineStage.Disassemble);

If anyone has an example of doing this, or can point me in the right
direction, it would be greatly appreciated.

Thanks in advance,

Jim

Jim,

> I am trying to create an instance of a BizTalk Pipeline object from within
> a
> .net method.
>
> The code would be something like this:
> Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMes sages pipeline =
> PipelineFactory.CreateEmptyReceivePipeline();
> pipeline.AddComponent(new BtfDasmComp(), PipelineStage.Disassemble);
>
> If anyone has an example of doing this, or can point me in the right
> direction, it would be greatly appreciated.

Why are you trying to create the pipeline components manually inside the
orchestration? You're supposed to create your custom pipeline in the
pipeline designer and then use that instead.

What are you trying to achieve this way?
(BTW, it looks like you're trying to mix code from two different samples
that use two different/unrelated object models... it won't work).

On 16-Jul-2006, "Tomas Restrepo \(MVP\)" <tomasr> wrote:

> Tomas

Hi Tomas,

Thank you for taking the time to reply. Please try and take the time to read
this. The resulting solution, if you can assist, will be a really elegant
one!

I know that there are other ways of doing this, e.g. multiple pipeline
disassembly stages, multiple receive locations i.e. pipelines and schemas),
does none address my particular requirement as fully as this solution would

I am trying to do something quite different from a .net component being
called by BizTalk.

I have a number of different feeds with different schemas.
I have an orchestration that calls the Receive Pipeline and loops round each
message in the interchange.

For each message I call the same .net component, which selects the correct
Flat File Schema and use it to valid the message.

The splitting of the different flat file interchange files and passing of
the individual records to the .Net component contained in the loop works
fine.

PROBLEM:
I need to work out how apply a Flat File Schema to a single flat file line
extracted from the file within a .Net component.
I need to apply the correct schema to each of the flat file lines passed to
the method from the loop.

WHAT AM I TRYING TO ACHIEVE:

I want a SINGLE orchestration that can validate 30 different flat file
formats. The orchestration creates a file containing all of the valid lines
contained in the input file and a file containing all of the invalid lines
rejected when the schema was applied in the .net component. In also creates
a detailed problem report containing each on the schema violation associated
with each of the flat file lines.

So basically if the input file contained:
ID,Name
1, Sid
2, Jim
3, Jill

I would use a generic schema to break the flat file into individual lines,
e.g. “2,Jim”.
Pass that to a .net method, which would workout which schema should be
applied to that flat file and then apply it within a try catch block. An
event handler is then called for EACH of the violation associated with the
applied scheme. This enables me to construct a details report on a line by
line basis.



I think I am heading in the right direction, but I am struggling with the
creation of the required objects from within .Net because they require
parameters such as IBaseMessageFactory and IPipelineContext.


// Create the flat file Disassembler, assign it the message to use.
Microsoft.BizTalk.Component.FFDasmComp ffdcomp = new FFDasmComp();
IPipelineContext pctx;

ffdcomp.InitNew();
ffdcomp.Disassemble(pctx, bmMessage);
IBaseMessage msg2 = ffdcomp.GetNext(null);


Does anyone know how to access this in the orchestration so I can pass it
into the .net component from the orchestration.
E.g. CallMySchemaApplyingMrthod(IPipelineContext pctx, IBaseMessageFactory
bmf);


Thanks in advance,

Jim

Hi Tomas,

I am just icluding some additional information about why I need to creation
a pipeline within a .net method.

If I was validating XML I could just use the standard xml validation methods
available in the .net classes. But because I need to validate a flat file
line, e.g. “1, Sid”, I need to run it through the flat file dissembler
component which is normally run by the pipeline component.

Because the flat file dissembler component requires IPipelineContext, etc,
I need to instantiate a pipeline first.

Regards,

Jim

Jim,

I think I get what you're trying to do, but I'm not 100% it *is* the best
way (in particular, I think performance won't be very good).

As I see it, you have two options:

1- Create a custom disassembler component. While writing a custom
disassembler is somewhat painful, you'll still be able to reuse the
FFDasmComp to implement it, which makes it somewhat easier. Furthermore,
you'll be able to use it by simply putting it into a regular custom pipeline
and invoking that from your orchestration (if that is indeed what you want;
though I see little reason why you wouldn't want to do it just straight on
the receive pipeline at this point).
Possibly Gile's sample here would be helpful:
http://blogs.msdn.com/gzunino/archiv...19/217277.aspx

2- Go on the way you're doing and dynamically create either pipelines or
directly invoke the FFDasmComp from your .NET component (which it seems
you're trying to do). The problem here is that the object models for
pipelines and orchestrations is very different, particularly in how it
represents messages (i.e. IBaseMessage vs. XLANGMessage and friends). This
is exactly part of the impedance mismatch that the library for calling
pipelines from orchestrations in BTS06 solves, as well as ensuring that the
pipeline gets an execution context compatible with the calling
orchestration. You can problably work around it, but it will take some work
to solve, particularly if you care about performance. Possibly your best
bet here would be to poke around the biztalk assemblies using Reflector to
see the kind of work they do (warning: you won't be able to do it the same
way, they rely on internal functionality not exposed to outsiders). You
might also get some ideas about how it works from looking around my
PipelineTesting library:
http://www.winterdom.com/weblog/2006...raryPart1.aspx

Hi Tomas,

Thank you for your response.

I think you do understand what I am trying to do now. I do understand the
performance implications but the flat files only contain a few hundred lines
and they are only received a few times a day, so this approach would be
suitable.

I understand the reason why you think is it a unusual approach, but it is
the only one that I know of that enables me to meet my objective:

1. Create two file from the input file, one containing all of the valid
lines and one creating all of the invalid lines. This seems a normal
requirement to me. They don’t want to receive emails/WSS on a line by line
basis. They can then send them the person who sent the file the file
containing all of the invalid lines so they can fix them
2. A report containing the invalid line followed by ALL of the problems
associated with that line.

I feel that BizTalk 2006 should be flexible enough to enable you to achieve
the above. I do not like the way we are force to be limited to what error
information it thinks you require.
E.g. If I had a line that contain 5 schema violations. In .net I would
associate a callback method and then apply the Schema to the line
(XmlDocument). The callback method would be called each time a violation
occurred. So you could end up with the following report:
Line 10: ID, Name, Age
1, Sid, gr,
Errors:
Name should be more that one character
Age must be numeric


Below shows the code that achieves this. It works fine! If I pass it
something like:
<ID>1</ID<Name>Sid</Name><Age>7</Age>
So why don’t I use it then? Enter the problem; I need to pass it something
like the FFDasmComp component applies a schema/s to:
1,sid,7

The .Net code below cannot validate a non XML document. This is why I need
to use the pipeline flat file disassembly component (FFDasmComp) to convert
the line ‘1,sid,7’ into an XML document
‘<ID>1</ID<Name>Sid</Name><Age>7</Age>’ and the validate it. If there is
another why of applying the flat file schema generated by the wizard to the
line ‘1,sid,7’ I would be interested in knowing how.


XmlReaderSettings settings = new XmlReaderSettings();
settings.Schemas.Add(null,SchemaFileLocation);
settings.ValidationEventHandler += new
ValidationEventHandler(ValidationCallback);
settings.ValidationFlags = settings.ValidationFlags |
XmlSchemaValidationFlags.ReportValidationWarnings;
settings.ValidationType = ValidationType.Schema;

// Extract the raw data line from the XML document
XmlDocument xmlDocDataLine = new XmlDocument();
xmlDocDataLine.LoadXml(sDataFeedLine);
XmlNode xnLine = xmlDocDataLine.SelectSingleNode("//Line");
sRawDataLine = xnLine.InnerText;

XmlTextReader xtrReader = new XmlTextReader(new
StringReader(sDataFeedLine));
xtrReader.WhitespaceHandling =
System.Xml.WhitespaceHandling.None;
xtrReader.Read();

XmlReader reader = XmlReader.Create(xtrReader, settings);
XmlDocument document = new XmlDocument();

document.LoadXml(sDataFeedLine);


public static void ValidationCallback(object sender,
ValidationEventArgs args)
{
string sss = "Failure. Validation was not successful.�
}




I have already looked at your code and used it as a template. The only
problem with the PipelineTest program is that I cannot find the correct
reference to add for the PipelineFactory object. I get a compilation error on
the following line:

Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMes sages pipeline =
PipelineFactory.CreateEmptyReceivePipeline();


If there is another why of applying the flat file schema generated by the
wizard to the line ‘1,sid,7’ I would be interested in knowing how.

Thanks in advance

Jim


"Tomas Restrepo (MVP)" wrote:
[..]

> Thank you for your response.
>
> I understand the reason why you think is it a unusual approach, but it is
> the only one that I know of that enables me to meet my objective:

I'm not arguing your objectives are wrong; they seem just fine to me and
worthwhile. I'm arguiing against the path you're choosing to follow :)

> I have already looked at your code and used it as a template. The only
> problem with the PipelineTest program is that I cannot find the correct
> reference to add for the PipelineFactory object. I get a compilation error
> on
> the following line:
>
> Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMes sages pipeline =
> PipelineFactory.CreateEmptyReceivePipeline();

That's why I say you're mixing two things here:
PipelineFactory.CreateEmptyReceivePipeline() is part of the code I *wrote*
for my PipelineTestingLibrary; it's nothing you get for free out of biztalk,
and it most certainly doesn't return an instance of
ReceivePipepelineOutputMessages but of yet another type I had to create
called ReceivePipeline.

As I said, if you choose to continue to try and do it this way, you won't
get it as easy. When I said go and look at the code I wrote, I was meaning
look at all the things I had to mock to get something usable (and I'm
leaning on a bunch of mocks microsoft itself created to create
pipeline.exe). This is because the Microsoft.XLANGs.Pipeline library is not
extensible in that way (as far as I know) and it relies on a bunch of
internal support classes (even calling internal methods in other libraries,
btw). As I said, it won't be easy and you won't get away with it with a
couple of lines of code ;)

Anyway, I still believe what you really want is a custom disassembler, but
if you go the other way, please keep in mind what I'm saying and spend some
time using Reflector on the Microsoft.XLANGs.Pipeline library. It's the only
way you'll understand how it all goes.

Another way of achieving your aim is to use a map with custom Xslt.
You can implement a Schematron pattern validating each line and producing
output
There is a short example here:
http://www.biztalkgurus.com/forums/viewtopic.php?t=1341

The Schematron language allows you to create a bunch of xpath rules for a
document. The default implementation is to use Xslt to transform the
Schematron rules file into Xslt which can then be applied against a
document. I have cut out the Schematron rules file and just produced the
resultant Xslt file.
This approach allows you to create context sensitive rules, like
if <node1> = 27 then <node2> must exist otherwise <node2> must not exist.
You cannot use Xml Schema for this type of validation.

I have implemented this to produce a complete error list for a document, you
could also use this approach to split good and bad records into two sections
and then split the document after the map.

Greg

"Tomas Restrepo (MVP)" <tomasr> wrote in message
news:3564
[..]

Hi Tomas,

Thanks for the response.
It is a shame I cannot implement the affect I desired.
E.g.
1. Create two file from the input file, one containing all of the valid
lines and one creating all of the invalid lines. This seems a normal
requirement to me. They don’t want to receive emails/WSS on a line by line
basis. They can then send them the person who sent the file the file
containing all of the invalid lines so they can fix them
2. A report containing the invalid line followed by ALL of the problems
associated with that line.

If I do implement the solution above, I will post to the group again and
make the solution (code) available to you so you can take a peek.

If I go down the normal route or create a customer pipeline. How close can I
get to the above solution?

If I use the normal approach, i.e. a different receives location for each
flat file and a schema that breaks the interchange into individual messages.
If an error occurs the message is routed to an error handing orchestration,
which writes it into a db or WWS. Then have another orchestration that looks
into the db periodically and groups the valid items into a single file and
also groups all the one marked as invalid into a single file.

Obviously I cannot collect any where near the level of schema exception
details to ensure each of the problems associated with each of the lines are
resolved before a re-run occurs.

The above required three separate orchestrations, db CPU, etc and still
doest achieve a superior result. In fact in is inferior because there are
more piece to the puzzle, and therefore more things to go wrong!

All because I cannot apply the FFDasmComp with a schema inside a .Net
component.

Questions
Do you have a better way of achieving two separate file (one good and one
bad) and an error report?

Another reason for calling the receive pipeline was so I new when the
orchestration had finished processing the flat file.

Is there any other way of determine when a flat file has finished being
processes?

I am just going to have an orchestration check the DB table and if nothing
has been added since the last record written for 5 minutes it will assume
that the file processing is completed and creation the 2 separate files.



"Tomas Restrepo (MVP)" wrote:
[..]

Hi Greg,

Thank you very much for your input.
I will look forward to looking into this possible solution. I will post the
results tomorrow.

Thanks mate.

Is sent this to Thomas:

It is a shame I cannot implement the affect I desired.
E.g.
1. Create two file from the input file, one containing all of the valid
lines and one creating all of the invalid lines. This seems a normal
requirement to me. They don’t want to receive emails/WSS on a line by line
basis. They can then send them the person who sent the file the file
containing all of the invalid lines so they can fix them
2. A report containing the invalid line followed by ALL of the problems
associated with that line.

If I do implement the solution above, I will post to the group again and
make the solution (code) available to you so you can take a peek.

If I go down the normal route or create a customer pipeline. How close can I
get to the above solution?

If I use the normal approach, i.e. a different receives location for each
flat file and a schema that breaks the interchange into individual messages.
If an error occurs the message is routed to an error handing orchestration,
which writes it into a db or WWS. Then have another orchestration that looks
into the db periodically and groups the valid items into a single file and
also groups all the one marked as invalid into a single file.

Obviously I cannot collect any where near the level of schema exception
details to ensure each of the problems associated with each of the lines are
resolved before a re-run occurs.

The above required three separate orchestrations, db CPU, etc and still
doest achieve a superior result. In fact in is inferior because there are
more piece to the puzzle, and therefore more things to go wrong!

All because I cannot apply the FFDasmComp with a schema inside a .Net
component.

Questions
Do you have a better way of achieving two separate file (one good and one
bad) and an error report?

Another reason for calling the receive pipeline was so I new when the
orchestration had finished processing the flat file.

Is there any other way of determine when a flat file has finished being
processes?

I am just going to have an orchestration check the DB table and if nothing
has been added since the last record written for 5 minutes it will assume
that the file processing is completed and creation the 2 separate files.



"Greg Forsythe" wrote:
[..]

Hi Jim,

> Thanks for the response.
> It is a shame I cannot implement the affect I desired.
> E.g.
> 1. Create two file from the input file, one containing all of the valid
> lines and one creating all of the invalid lines. This seems a normal
> requirement to me. They don't want to receive emails/WSS on a line by line
> basis. They can then send them the person who sent the file the file
> containing all of the invalid lines so they can fix them
> 2. A report containing the invalid line followed by ALL of the problems
> associated with that line.
>
> If I do implement the solution above, I will post to the group again and
> make the solution (code) available to you so you can take a peek.
>
> If I go down the normal route or create a customer pipeline. How close can
> I
> get to the above solution?

As close as you want. There's no requirement that a custom disassembler
submits to the message box the same message (semantically speaking) it
received. You could, if you wanted, have it generate multiple messages as
you needed or generate a completely custom messsage containing your error
report as an XML stream or whatever you want. You need to be careful about
it, but it's certainly possible.

Note I'm not talking about a custom pipeline, I'm talking about actually
creating a custom disassembler component.

[..]
> details to ensure each of the problems associated with each of the lines
> are
> resolved before a re-run occurs.
>
> The above required three separate orchestrations, db CPU, etc and still
> doest achieve a superior result. In fact in is inferior because there are
> more piece to the puzzle, and therefore more things to go wrong!
>
> All because I cannot apply the FFDasmComp with a schema inside a .Net
> component.

You *can* apply it, you just have to work for it.

>
> Questions
> Do you have a better way of achieving two separate file (one good and one
> bad) and an error report?
>
> Another reason for calling the receive pipeline was so I new when the
> orchestration had finished processing the flat file.

Don't see what the problem is, just have the orchestration send a message to
signal it endedd processing.

Hi Tomas,

How would the orchestration determine when the flat file has finished being
processed?
i.e. the interchange is being broken into single messages. So, multiple
instances of that orchestration may be being run on a number of the single
messages split out of the flat file.

How could any of the orchestration instances know when processing have been
completed?

Regards,

Jim


"Tomas Restrepo (MVP)" wrote:
[..]

Jim,

> How would the orchestration determine when the flat file has finished
> being
> processed?

What do you need this information for?

> i.e. the interchange is being broken into single messages. So, multiple
> instances of that orchestration may be being run on a number of the single
> messages split out of the flat file.

You can also *not* brake it during disassembly and instead do the debatching
inside the orchestration, if that suits you.

Hi Tomas,

Thank for the response.

>> What do you need this information for?

Any schema violations will cause the line/message to be suspended, which I
will use error routing so an orchestration can receive the message and write
it to a DB.

So, once the file has been processed I want to create a file containing all
of the lines with errors and what the problem was. Although I could create
the file immediately in the error handler and append to it, my scenario is
more complicated (e.g. many different file formats, need to associate
additional information in error handler so I can determine which item belongs
to what batch file, …)

>> You can also *not* brake it during disassembly and instead do the debatching
inside the orchestration, if that suits you.

Yes sure, this is what I will probable do so I can determine when it has
finished. Obviously this affects performance. It is a shame this approach has
to be adopted JUST to determine when the file has been processed.

I would like have continued with my initial approach, but as I would have to
use reflection to access internal methods etc it is not a suitable solution,
i.e. a service patch may update the internal method interfaces.

If you or any one can think of a better way on processing a flat file
interchange and creating a file containing valid lines and a file containing
invalid lines and an line by line error report please let me know.

Thanks again,

Jim


"Tomas Restrepo (MVP)" wrote:
[..]