Deserialize nested Array property with Stream in JSON.net

Sometimes, you have to deal with importing big data files and the data is not really what you’ve expected. Here is a solution for handling Deserializing data with a nested property (array). You may need to add a Depth parameter if propertyName is not enough to identify the field to deserialize.
Of course, the method use yield in order to not allocate any memory, very handy for big files !

Enjoy 🙂

/// <summary>
/// Deserialize nested array property as IEnumerable
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="stream">Stream</param>
/// <param name="propertyName">nested property to deserialize</param>
/// <param name="settings">JSON.net setting</param>
/// <returns>Enumerable of T</returns>
public static IEnumerable<T> DeserializeEnumerableNestedProperty<T>(Stream stream, string propertyName, JsonSerializerSettings settings = default)
{
	using StreamReader sr = new StreamReader(stream, leaveOpen: true);
	using JsonTextReader reader = new JsonTextReader(sr);

	// looking for property
	while (reader.Read())
	{
		if (reader.TokenType != JsonToken.PropertyName)
		{
			continue;
		}

		var currentPropertyName = reader.Value as string;

		if (currentPropertyName?.Equals(propertyName, StringComparison.OrdinalIgnoreCase) != true)
		{
			continue;
		}

		break;
	}

	// deserialize Array as enumerable
	while (reader.Read())
	{
		if (reader.TokenType != JsonToken.StartArray)
		{
			continue;
		}

	        var serializer = Newtonsoft.Json.JsonSerializer.CreateDefault(settings);

		while (reader.Read() && reader.TokenType == JsonToken.StartObject)
		{
			T obj = serializer.Deserialize<T>(reader)!;
			yield return obj;
		}

		break;
	}
}
Posted in Non classé | Leave a comment

Optimize IN clause with LINQ Entity Framework with Ranges

Problem

Suppose you have this simple query :

var ids = new long[]{1,3,4,5,8,9,10,78,34,23...};
Context.MyTable.Where(a=>ids.Contains(a)).ToList()

if ids contains one million number, Linq provider will write a giant query like :

SELECT id, field1, field2 WHERE
id IN (1,3,4,5,8,9,10,78,34,23 ...) // until one million numbers

You will have an exception because it’s too much for the SQL provider to execute this query.
You’ll probably think about these 2 solutions :

  • Inserting all theses datas in a temporary table (with a third party library), then joining
  • Splitting your queries in multiple chunks

Both are bad solutions because you have to Upload millions of number to the SQL Server.

Solution

To resolve this problem, you have to group the numbers by ranges.

Range 0 : 1,2,3,4,5
Range 1 : 19,20
Range 2 : 45,46,47,48,49,50
…..
Range 10 : 125558,125559,125560,125561,125562

IList<Range> GroupArrayToRanges(IEnumerable<int> values)
{
    var res = values.Order().Select((num, index) =>
    new
    {
        Id = num,
        Rank = index,
        Group = num - index
    })
    .GroupBy(a => a.Group)
    .Select(a => new Range(a.Min(a => a.Id), a.Max(a => a.Id)))
    .ToList();
    return res;
}

Now you have the groups, you need to rewrite the correct LINQ query.

var ids = new int[]{1,3,4,5,8,9,10,78,34,23...};
var ranges = GroupArrayToRanges(ids);
Context.MyTable.Where(a=>
   (a>=ranges[0].Start && a<=ranges[0].End) 
|| (a>=ranges[1].Start && a<=ranges[1].End)
|| (a>=ranges[2].Start && a<=ranges[2].End)
...
|| (a>=ranges[10].Start && a<=ranges[10].End)  
).ToList()

Building this query dynamically would need the use of Expression Trees because of logical OR.
If we want to build this query without Expression, we need to convert to an AND query, by including the whole range first and excluding ranges like this :

var ids = new int[]{1,3,4,5,8,9,10,78,34,23...};
var ranges = GroupArrayToRanges(ids);
Context.MyTable.Where(a=>
   (a>=ranges[0].Start && a<=ranges[10].End) // include whole range
&& !(a>ranges[0].End && a<ranges[1].Start) // exclude
&& !(a>ranges[1].End && a<ranges[2].Start) // exclude
&& !(a>ranges[2].End && a<ranges[3].Start) // exclude
...
&& !(a>ranges[9].End && a<ranges[10].Start)  
).ToList()

Now you can enjoy an elegant solution without uploading tons of data.
The only drawback of the solution is it will not work if your data cannot be grouped by interval like 1,3,6,8,12,14,16,29,31 => it will create 9 groups…
But if you have 1,2,3,4…..1001045 data, that will perfectly fit !

Go further with SQL and Linq

This is the equivalent way to do in SQL

SELECT 
     MIN([t].[Id]) AS [Min], 
     MAX([t].[Id]) AS [Max]
FROM (
    SELECT [s].[Id], [s].[Id] - (
        SELECT COUNT(*)
        FROM [SuperTable] AS [s0]
        WHERE [s0].[Id] < [s].[Id]) AS [Key]
    FROM [SuperTable] AS [s]
) AS [t]
GROUP BY [t].[Key]

-- if you really need performance, you would better go with ROW_NUMBER

SELECT 
     MIN([t].[Id]) AS [Min], 
     MAX([t].[Id]) AS [Max]
FROM (
    SELECT 
	[s].[Id], 
	s.Id - (ROW_NUMBER() OVER(ORDER BY id ASC)) AS [Key]
    FROM [SuperTable] AS [s]

) AS [t]
GROUP BY [t].[Key]

-- if you have an old MYSQL server, you can emulate ROW_NUMBER with a session variable
SET @row_number := 0;
SELECT 
     MIN(t.id) AS Min, 
     MAX(t.id) AS Max
FROM (
    SELECT 
	s.id, 
	s.id - s.RowNumber as `Key`
    FROM (    
	     SELECT id,  ((@row_number := @row_number + 1)) as RowNumber
		FROM
		T_USERS
        ORDER BY id
    ) AS s   
) AS t
GROUP BY t.Key;

And the Linq (SQL transcriptable) Syntax (without ROW_NUMBER function as it’s not supported by current LINQ providers):

var q = from s in context.Table
        orderby s.Id
        select new
        {
            Id = s.Id,           
            Rank = (from o in context.SuperTables where o.Id < s.Id select o).Count(),
            Group = s.Id - (from o in context.SuperTables where o.Id < s.Id select o).Count(),
        } into sub2
        group sub2 by sub2.Group into sub3
        select new
        {
            Min = sub3.Min(a => a.Id),
            Max = sub3.Max(a => a.Id)
        };

In a non SQL context, you would probably rewrite Select by using Select((s,index)=> syntax to perform ranking like ROW_NUMBER, it’s more efficient.

int[] data = new int[] { 1, 2, 3, 4, 5, 7, 8, 10, 11, 12 };

var res = data.Order().Select((num, index) =>
    new
    {
        Id = num,
        Rank = index,
        Group = num - index
    })
    .GroupBy(a => a.Group)
    .Select(a =>
    new
    {
        Min = a.Min(a => a.Id),
        Max = a.Max(a => a.Id)
    })
    .ToList();

Update : Use ROW_NUMBER within Entity Framework Core

I developed my own extension to support Row_Number within EntityFramework Core for SqlServer,Sqlite,Mysql,PostreSql here :

https://www.nuget.org/packages?q=Webrox.EntityFrameworkCore
Github repository : https://github.com/Poppyto/Webrox.EntityFrameworkCore

Then use it like that in order to get the id intervals :

 var res = context.Table
.Select((a, index) =>
 new
 {
	 Id = a.Id,
	 Group = a.Id - EF.Functions.RowNumber(EF.Functions.OrderBy(a.Id))
 })
 .GroupBy(a => a.Group)
 .Select(a =>
 new
 {
	 Min = a.Min(a => a.Id),
	 Max = a.Max(a => a.Id)
 })
 .ToList();
Posted in Non classé | Tagged , , , , , | Leave a comment

TextBox Autocomplete – Crash AccessViolationException – Workaround

It seems a nogo, but there’s a bug with AutoCompletion on Winforms since years. (.net7 here)
The only way to make it work is to set the source only once.
If you try to have a dynamic Suggestion, you could set :

Text_OnChanged(object e)
{
AutoCompleteStringCollection.AddRange("hello");
// CRASH AccessViolationException
}

It will crash with AccessViolationException.

Based on my Winforms TextBox.cs reading, I found an elegant solution.

1. Set private field _fromHandleCreate to true after HandleCreated

public class MyTextBox : TextBox
{
    FieldInfo? fiFromHandleCreate = typeof(TextBox).GetField("_fromHandleCreate",
                                                        BindingFlags.Instance |
                                                        BindingFlags.NonPublic |
                                                        BindingFlags.SetField |
                                                        BindingFlags.GetField);

    protected bool IsFromHandleCreate
    {
        get => (bool)fiFromHandleCreate.GetValue(this);
        set => fiFromHandleCreate.SetValue(this, value);
    }

    protected override void OnHandleCreated(EventArgs e)
    {
        base.OnHandleCreated(e);
        IsFromHandleCreate = true; //important to not destroy AutoComplete handle, avoid flickering 
    }
}

2. Redefine AutoCompleteStringCollection

The main trick is there’s always at least one element : String.Empty, because when the collections is cleared, Autocompletion is disconnected from the Textbox (and recreation bring bug)
The second trick is to set RemovedItem to String.empty instead of removing them
The third trick is to occupate empty Slot when Adding to reduce memory consumption

    /// <summary>
    /// Make AutoCompleteStringCollection dynamic
    /// </summary>
    public class MyAutoCompleteStringCollection : AutoCompleteStringCollection
    {
        static readonly string EmptyEntry = String.Empty;//space magic string
        public MyAutoCompleteStringCollection()
        {
            this.Add(EmptyEntry);//never delete this entry
        }

        FieldInfo? fidata = typeof(AutoCompleteStringCollection).GetField("data",
                                                        BindingFlags.Instance |
                                                        BindingFlags.NonPublic |
                                                        BindingFlags.SetField |
                                                        BindingFlags.GetField);
        /// <summary>
        /// base.data private field
        /// </summary>
        protected ArrayList Data
        {
            get => (ArrayList)fidata.GetValue(this);
            set => fidata.SetValue(this, value);
        }

        /// <summary>
        /// base.Clear is Evil => it disconnect the Autocompletion....:(
        /// </summary>
        public new void Clear()
        {
            var vData = this.Data;
            vData.RemoveRange(1, this.Count-1);

            //notify
            if (this.Count>1)
                base.OnCollectionChanged(new System.ComponentModel.CollectionChangeEventArgs(System.ComponentModel.CollectionChangeAction.Refresh, null));
        }

        /// <summary>
        /// Remove but without removing the String.Empty rentry
        /// </summary>
        /// <param name="value"></param>
        public new void Remove(string value)
        {
            int index = IndexOf(value);
            if (index > 0) //keep String.Empty
                base[index] = string.Empty; // clean slot + notify
        }
        /// <summary>
        /// RemoveAt but without removing the String.Empty rentry
        /// </summary>
        public new void RemoveAt(int index)
        {
            if (index > 0)//keep String.Empty
                base[index] = string.Empty; // clean slot + notify
        }

        /// <summary>
        /// Add to an unoccuped slot or a new one
        /// </summary>
        public new void Add(string text)
        {
            int index = -1;
            for (int i = 1; i < Count; i++)
            {
                if (this[i] == string.Empty)
                {
                    index = i;
                    break;
                }
            }
            if (index > 0)
                base[index] = text; // occupate the slot + notify            
            else
                base.Add(text); // add+notify
        }

        /// <summary>
        /// AddRange with efficient slot occupating
        /// </summary>
        /// <param name="values"></param>
        public new void AddRange(string[] values)
        {
            var vData = this.Data;

            int cpt = 0;
            for (int x = 0; x < values.Length; x++)
            {
                int index = -1;
                for (int i = 1; i < Count; i++)
                {
                    if (vData[i] == string.Empty)
                    {
                        index = i;
                        break;
                    }
                }
                if (index > 0)
                {
                    vData[index] = values[x]; // occupate the slot                
                    cpt++;
                }
                else
                    break;
            }

            //Send last part via AddRange
            if (cpt < values.Length)
                vData.AddRange(values.Skip(cpt).ToArray());

            if(values?.Length>0) //notify
                base.OnCollectionChanged(new System.ComponentModel.CollectionChangeEventArgs(System.ComponentModel.CollectionChangeAction.Refresh, null));

        }

    }

Et voilà !
Stephane

Posted in Non classé | Tagged , | Leave a comment

The solution for validating an email address (RFC compliant) in .net C#

It’s a common question : “How can I validate one email address” ?

There are 0b10 (2) types of developers : lazy and ignorant.

  1. Lazy people always wins, because they do not reinventing the wheel, they prefer to search already done API.
  2. Ignorant people will probably use Regexp (I love regexp but not in this case) or write a new algorithm, they could spend few hours or days to implement the full RFC, good luck !

So the code is obvisouly server side, and in .net :

try
{
   new HttpRequestMessage().Headers.From = "mail@foo.com";
   //here the mail is valid
}
catch(FormatException ex)
{
// here the email is invalid
}
Posted in Non classé | Leave a comment

WinRT Apps : Slow Compilation when you add WinRT components : Solution

PROBLEM

When you create a Blank WinRT project (W8 or WP8.1) with Visual Studio, if you compile as it, it takes : 2 seconds => great !

BUT, if you just add a simple WinRT component library (winmd) as reference and build, it takes 12 to 15 seconds… => wt….???! (if you add a DLL, there is no problem)

Well, it’s not a good start for a project, I didn’t find anything on the Internet about this issue, but I’m sure some developers will burn their computer if they have this issue (maybe they already bought a new CPU ;) ?)

WHY ?

If you verbose the MSBuild log, you can see this line

> 10179 ms GenerateAppxManifest 1 calls

Ok GenerateAppXManifest is the CPU eater ! We have to found the solution…

SOLUTION

The Solution I found is to override the value AppxHarvestWinmdRegistration to false for MSBUILD in your App csproj, and to fill your AppxManifest .xml Extensions section as it was generated before in the Debug/Release directory – before this hack :) -.

1. open your project.csproj, and add :

<PropertyGroup>   
<AppxHarvestWinmdRegistration >false</AppxHarvestWinmdRegistration> 
</PropertyGroup> 
<Import Project="$(MSBuildExtensionsPath)\Microsoft\WindowsXaml\v$(VisualStudioVersion)\Microsoft.Windows.UI.Xaml.CSharp.targets" />

2. Then, you have to update your AppxManifest.xml like it was before in the Debug/Release dir for the Extensions section (specially if you use the Background Audio or anything which communicate with the external process) :

<Package....

<Extensions>
 <Extension Category="windows.activatableClass.inProcessServer">
 <InProcessServer>
 <Path>CLRHost.dll</Path>
 <ActivatableClass ActivatableClassId="BackgroundAudioTask.MyBackgroundAudioTask" ThreadingModel="both" />
 </InProcessServer>
 </Extension>
 </Extensions>

Have a nice coding day !

@poppyto

Posted in Non classé | Leave a comment

Solution to Empty Emulator List (plus 0x80070002) in Windows Phone SDK 8

Sometimes, Visual Studio fails and you can’t do anything. You can repair or reinstall but in this case it’s not works.

PROBLEM : You can’t see any Emulators on Visual Studio :

and when you try to compile it throws 0x80070002  (File not found exception) :

SOLUTION :

So there are missing files, I catch them with ProcMon and by using MultiTargetingConnectivity API (this is THE api which throws 0x80070002) They are here in this location C:\Users\[Your User]\AppData\Local\Microsoft\Phone Tools\CoreCon\11.0

They are :
– Microsoft.ServiceCategories.xsl,
– Microsoft.VisualStudio.ServiceCategories.xsl
– Microsoft.XDE.ServiceCategories.xsl
 Microsoft.TypeMaps.xsl

It can be difficult to retreive these files (I repaired reinstall SDK 8 with no luck, trying to remove 11.0 directory and reinstall etc) so as I’m a nice guy, you could find the directory 11.0 zipped here : Download the full directory 11.0, extract here and copy/paste the missing file in your directory.

Have a nice coding day :)
Stephane

Posted in Non classé | Leave a comment

Solution for blurry rendering with DirectX11 on Windows Phone 8

If you experienced a global blurry rendering on your screen when you use DirectX11 on your Windows Phone 8, you probably fails a thing :

The swapchain size must be the screen size (means portrait), not the landscape size.

So  :

swapChainDesc.Width = 800;//
swapChainDesc.Height = 480; // blurry rendering !

swapChainDesc.Width = 480;//
swapChainDesc.Height = 800;  // perfect rendering !

That’s all, take care !1

Posted in Non classé | Leave a comment

Solution for Error ERROR_DLL_INIT_FAILED while Loading winmd file in Windows RT

If you experienced the Exception : Error in the DLL (Exception from HRESULT: 0x8007045a (ERROR_DLL_INIT_FAILED))

The solution is simple : You have to remove the main function.
(I know, why the fonction “main” is called with DLL? idk :))

Posted in Non classé | Leave a comment

Solution for Error CO_E_ERRORINDLL while Loading winmd file in Windows Phone 8

If you experienced the Exception : Error in the DLL (Exception from HRESULT: 0x800401F9 (CO_E_ERRORINDLL))

The solution is simple :

You Must Add the _WINRT_DLL flag in the compiler definitions.

Posted in Non classé | Leave a comment

Windows Phone Store 8 shows neutral titles instead of localized titles

Dear you,

I experienced a bug in the Windows Phone 8 Store : I localized my app titles with the AppResLib.dll (& mui) and the tiles title were ok in the phone but the WP8 Store shown only the neutral language title. (on WP7 Store only, it looks ok!)

SOLUTION

In the WMAppManifest.xml (WP8) or in the project settings (WP7), you have to uncheck generic language codes (as “fr”,”it”,”de”) and check all the sub-language codes like “fr-FR”,”fr-BE”,”it-IT”, etc. If you check both generic and sub-languages, this will not working.

CONCLUSION

For the moment : don’t ever use the generic language codes (if you used them in your old WP7 app, this will appears Ok on WP7 Store but incorrectly on WP8 Store)

That’s all and this works !

Complete thread here.

Posted in Non classé | Leave a comment